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(57) Abstract 

A digital telemedicine or other remotely-controlled imaging system 
for enabling an operator at a local site (14) to analyze images (70) 
received in digital form from an auto-focus microscope located at a 
remote site (12). The system (10) includes a bandwidth optimization 
system and a virtual focus system with a graphic user interface (50). The 
bandwidth optimization system enables an operator to convenientiy select 
the resolution of images to be observed, and to enhance the resolution 
of portions of images currently under observation. The computational 
burden at the remote site and transmission requirements between the sites 
are thereby reduced. The virtual focus system enables an operator to 
emulate direct and continuous control over the microscope focus. The 
focus system is convenient to use and enables the operator to effectively 
observe the specimen at a number of different focal planes. 
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DIGITAL TELEPATHOLOGY IMAGING SYSTEM WITH 
BANDWIDTH OPTIMIZATION AND VIRTUAL FOCUS 
CONTROL FUNCTIONS 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates generally to remotely-controlled imaging systems. 
In particular, the present invention is a remotely-controlled microscope imaging system 
for telepathology having programmed bandv^dth optimization and virtual focus control 
functions. 

Description of the Related Art 

Pathologists routinely assist surgeons by analyzing and providing diagnostic 
opinions on frozen sections of tissue samples removed from patients during surgery. 
Frozen sections are small tissue specimens which are quick-frozen, sliced, treated with 
dyes and placed on slides. The slides are then analyzed through a microscope by the 
pathologists. The diagnoses provided by the pathologists are used by the surgeons in the 
course of determining the nature and extent of further surgical procedures. For example, 
pathologists often analyze tissue samples to determine whether they are cancerous, and to 
determine the type of cancer present. Using this information the surgeon will determine 
which tissue should be removed from the patient. 

A typical turnaround time for preparing a frozen section is about fifteen minutes. 
During the tissue sample analysis session a pathologist will typically first rapidly scan the 
entire sample for gross features at a relatively low magnification, and identify particular 
sample regions of interest for more detailed study. The selected regions of interest are 
then observed at relatively high magnifications. It is typical for a pathologist to observe 
many separate regions at a number of different magnifications during an analysis session. 
The length of time required by the patholo^st to complete this procedure and provide a 
diagnosis often about five to seven minutes. 

During diagnostic sessions patholo^sts will often observe a given specimen 
region at a number of different focal planes. The need for these range of focus 
obser/ations is due at least in part to the fact that the specimen being imaged has depth 


-1- 


wo 99/13360 


PCT/US98/18852 


(i.e., is three-dimensional). Portions of the specimens at different depths must therefore 
be imaged at diflFerent focal planes. 

Large hospitals and universities often maintain an on-site clinical pathology 
laboratory and staflF of pathologists to provide these services on demand. Since the 
diagnoses can be used to guide procedures to completion during a single surgery, the 
immediate access to the pathologist's expertise during surgery can provide significant 
benefits in terms of both economic and recuperative aspects of healthcare delivery. 

Smaller and outlying hospitals often rely on periodic visits fi-om "circuit-riding" 
pathologists that provide scheduled service to a group of hospitals. This commuting 
arrangement can be inefficient, since certain types of surgeries must be scheduled to 
coincide with the pathologist's visit. Travel time can also be an inefficient use of the 
pathologist's time. Furthermore, if routine or emergency surgical procedures being 
performed when the pathologist is not scheduled suddenly require a pathologist's 
expertise, the tissue sample must be shipped to a remote laboratory for diagnosis. The 
patient may then have to undergo surgery yet another time after the pathological diagnosis 
has been completed. These procedures can result in increased costs and reduced 
effectiveness of healthcare delivery. 

A knovm solution to the problems associated with the lack of timely on-site 
access to the expertise of pathologists and other medical personnel is the application of 
telecommunications technology, also known as telemedicine. Telepathology imaging 
systems, a type of remotely-controlled imaging systems which use communication 
technology to enable the remote observation of medical specimens, are knovm, and 
generally fall into one of two categories. So-called static telepathology systems make use 
of a trained pathologist and a laboratory having a microscope vAih distal electronic 
imaging capabilities. Systems of this type are disclosed generally in the Weinstein et al. 
article Telepathology: Long-Distance Diagnosis, American Journal of Clinical Pathology, 
Vol 91, pp. S39 - S42, 1989. A clinician or other "sender" prepares the fi-ozen sample 
tissue specimen, analyzes the specimen through the microscope, and identifies a number 
of specimen regions that appear to be of interest. Digital images of the selected regions of 
interest are then generated, and electronically transferred to another site for analysis and 
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diagnosis by a pathologist. The pathologist can also communicate with the sender (e.g., 
by telephone) during these procedures, and request images of specific sample regions. 

Dynamic systems such as those disclosed in the Weinstein et al. U.S. Patent 
5,216,596 make use of a trained clinician and a laboratory having a microscope with a 
conventional analog video camera. "Live" images of the specimen sample produced by 
the video camera are transmitted to a television monitor using conventional NTSC or 
PAL analog video signal protocols, or compressed and transmitted using well-known 
digital video conferencing technologies for transmission to remote sites. 

Since the operator has remote control over the microscope focus function in 
systems such as those disclosed in the Weinstein et al. patent, he or she can observe a 
number of diflferent focal planes of the remotely-located specimen. Other remotely- 
controlled microscope systems, however, such as that described in the above-identified 
related application entitled "Digital Telepathology Imaging System" make use of 
conventional automatic focus systems on the remote microscope. Each time a new image 
re^on is imaged, or the magnification changed, the automatic focus system is actuated to 
focus the microscope. Digital data representative of a still image is then transmitted to the 
pathologist for observation. Since systems of this type do not provide pathologists with 
direct access to the focal plane adjustment of the microscope, the pathologists are unable 
to remotely observe diflferent focal planes of the image. 

Known technology for image and video transmission and virtual environment 
rendering typically allocates computational and transmission resources which vary based 
on image content, interfi*ame motion or scene complexity. Generally, the delivery of 
imagery with detailed content requires more encoding volume or computational eflfort 
(i.e., bandwidth) than imagery which has little detail or has predominately lowpass 
characteristics. 

For the transmission of discretized real-world imagery, the conventional (e.g., 
open-loop or machine-based) approach to determining which portions of an image 
sequence require extra bandwidth relies on a decomposition or blockwise analysis of the 
image sequence at the transmitter. Transmission and computational resources are 
thereby expended to decompose and prioritize an image sequence in terms of fi-equency 
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content (equivalently, correlation) and/or motion of image content between consecutive 
frames. 

During the use of remotely-controlled imaging systems, certain portions of an 
image under observation may be of higher interest than others. In telepathology, for 
example, if an area of a transmitted image has a dominant color such as blue, it is more 
likely to be an area of malignancy and therefore of greater interest to a pathologist than 
other (e.g., non-blue) portions of the image. Applications of this type generally demand 
lossless transmission, high spatial precision or color-space accuracy and relatively large 
computational resources. However, the prioritization of image data by conventional 
methods for encoding and transmission are generally not appropriate under these 
circumstances. 

It is evident that there is a need for improved remotely controlled imaging 
systems. In particular, there is a need for improved bandwidth prioritization systems for 
telepathology and other remotely controlled imaging systems. There also is a need for a 
mechanism by which an operator can obtain images at a number of different discrete focal 
planes from a microscope or other imag^ig instrument at a remote location. It would be 
especially advantageous if such a remote focusing system were able to emulate 
conventional direct control over the focal plane. To be commercially viable and 
eflScacious, the systems must be cost-effective and enable timely, high-quality healthcare 
delivery. A telemedicine imaging system providing these benefits should be capable of 
generating diagnostic-quality images while enabling real-time image selection control. 
The system should also make efficient utilization of communication channel and 
computational resources while using lossless image transfer protocols. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a diagrammatic illustration of a telepathology ima^ng system which 
includes bandwidth optimization and virtual focus systems in accordance Avith the present 
invention. 

Figure 2 is an detailed diagrammatic illustration of one of the server systems 
shown in Figure 1 , 
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Figure 3 is an illustration of the graphic user interface (GUI) menu generated and 
displayed by the client system shown in Figure 1. 

Figure 4 is an illustration of a GUI Open Session menu. 

Figure 5 is an illustration of a tiled slide overview specimen image. 

Figure 6 is an illustration of several cascaded specimen images vwth the GUI 
microscope control menu and zoom factor menu on the top image. 

Figure 7 is an illustration of several cascaded specimen images with the GUI 
image region selection reticle on the top image. 

Figure 8 is an illustration of GUI microscope control and resolution menus 
generated and displayed by the client system shown in Figure 1 . 

DETAH DESCRIPTION OF THE PREFERRED EMBODIMENTS 
System Architecture and Overview 

A "virtual microscope" imaging system 10 for telepathology (i.e., a remotely- 
controlled imaging system) which includes bandwidth optimization and virtual focus 
systems in accordance with the present invention can be described generally with 
reference to Figure 1 . As shown, system 10 includes one or more (three are shown) 
"server" systems 12 which are interfaced to a "client" system 14 over a digital 
telecommunication channel 16. Client system 14 is located at a hospital or other local site 
of a pathologist, and includes a workstation 18, monitor 20 and operator-actuated user 
input 21 such as keyboard 22 and/or mouse 24. Each server system 12 will typically be 
located at a hospital or other remote surgical site which does not have a staff pathologist, 
and includes a workstation 26, monitor 28, microscope station 30 and keyboard or other 
user input (not shown). During surgical procedures being performed at the remote site of 
a server system 12, a technician or other clinician will prepare slides of frozen sample or 
other tissue specimens and position the specimen slides on microscope station 30. A 
pathologist or other clinician at the local site of client system 14 remotely controls the 
operation of the server system 12 through the use a graphical user interface (GUI) 
displayed on monitor 20 and the operator-actuated user input 21. In response to digital 
server system control commands received from the client system 14 over the 
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communication channel 16, digital still images of the specimen are generated by the 
microscope station 30, transmitted to the client system and displayed in diagnostic-quality 
form on the monitor 20. The pathologist can, for example, observe an image of the entire 
specimen at a relatively low magnification, and identify particular specimen regions for 
more detailed analysis at greater magnification levels. Server system control commands 
such as specimen region conunands designating the selected specimen regions, 
magnification commands designating the desired magnification and illumination 
commands for controlling specimen illumination and imaging parameters of the camera 
are then generated by the client system 14 and transmitted to the server system 12 over 
the communication channel 16. Server system 12 processes the control commands, 
manipulates the specimen illumination and camera parameters, and transmits to the client 
system 14 specimen image data for the selected specimen region at the desired 
magnification. 

Communication channel 16 will typically be a broad bandwidth public network 
communication link capable of supporting packet-switched data communications. For 
example, ISDN (integrated services digital network) or Tl telephone lines can be used. 
Conventional and commercially available digital switching systems (not shown) can be 
used to interface seiver system 12 and client system 14 to the communication channel 16. 
Prototypes of system 10 were developed using an ATM-based (asynchronous transfer 
mode) optical channel 16 and an Ethernet network. 

A representative server system 12 can be described in greater detail with reference 
to Figure 2, Microscope station 30 is a conventional and commercially available 
instrument which includes a microscope 3 1 and a stage 34 for receiving and supporting 
specimen slides (not separately shown) to be imaged by lenses or objectives 36. The 
stage 34 is mounted to the base of the microscope 3 1 by an x-y stage drive 40 so the 
position of the slide with respect to the objectives 36, and therefore the region of the 
specimen being imaged, can be changed. The microscope 3 1 includes several objectives 
36, each of which provides a dififerent magnification power. Focusing is performed by a 
conventional automatic focus control system (not separately shown). The objectives 36 
are driven and positioned by a drive 38 to change the magnification of the microscope. 
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Although not shown, the microscope 31 also includes a conventional slide illumination 
light source and actuator-driven neutral density filter for controlling the specimen 
illumination level. 

A high-resolution camera 42 generates true-color image data representative of the 
specimen images produced by the microscope. This image data is digitized by the 
workstation 26. The camera 42 and digitizing system in workstation 26 function as a 
digital imaging system. Prototypes of system 10 include a camera 42 having a base 
resolution of at least approximately 600 x 800 pixels, and a digitizing system capable of 
capturing 8-bit per pixel (bpp) per color band still images (for a total of 24 bpp color). 
Conventional imaging control parameters of camera 42 can be accessed and controlled. 

Microscope station 30 is interconnected to workstation 26 through one or more 
interfaces illustrated generally at 44. Prototypes of system 10, for example, include an 
analog interface for connecting camera 42 to the workstation 26 and its digitizing system, 
and RS-232 busses for interconnecting the stage drive 40, objective drive 38, automatic 
focus system and image illumination system to the workstation 26. Monitor 28 is 
interconnected to workstation 26 by bus 46. These systems are capable of acquiring a still 
image and displaying that image at full resolution on the monitor 28 in less than about 1 
second. 

Workstation 26 is a conventional programmable computer, and is programmed 
with server system control software to control the operation of microscope station 30 in 
response to digital server system control commands received fi*om the client system 14. 
Control commands received fi-om server system 14 include camera commands, specimen 
region commands, magnification commands, focus commands and illumination 
commands. The specimen region commands include information designating a specific or 
desired region of interest on the specimen slide, and in one embodiment include 
information designating coordinates on the specimen slide in terms of "stage units." In 
response to the receipt of specimen region commands, workstation 26 actuates stage 
drive 40 and moves stage 34 to position the selected region of the specimen slide under 
objective 36. The magnification commands include information designating the desired 
magnification at which the selected specimen region is to be imaged. In response to the 
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receipt of magnification commands, workstation 26 actuates drive 38 and causes the 
selected objective 36 to be used to image the specimen region. In response to camera 
commands, camera 42 generates specimen image data representative of a still image of 
the specimen re^on being imaged by the microscope 3 1 . As described above, this image 
data is digitized by the workstation 26. 

As noted above, microscope 3 1 includes automatic focus and specimen slide 
illumination control systems. In response to focus commands received fi-om client system 
14, workstation 26 initiates an automatic focus routine by the automatic focus system of 
the microscope 3 1 . Workstation 26 also initiates an automatic focus routine each time the 
magnification is changed. Workstation 26 also controls the illumination system of the 
microscope 3 1 (e.g., the neutral density filter) to change the illumination level of the 
specimen slide in response to illumination commands received fi-om the client system 14, 
and can vary the imaging parameters of camera 42. 

Prototypes of system 10 use conventional TCP/IP protocol (transport control 
protocol/internet protocol) for the transfer of the server system control commands and 
specimen image data. The TCP/IP protocol of the prototype system 10 is supported by 
the Windows Sockets application programming interface (API) which provides 
handshaking, error reporting and associated fianctions, and is therefore compatible with 
local area network (LAN) as well as LAN-over-broadband telecommunications 
architectures. 

Client system 1 4 can include a workstation 1 8 and monitor 20 similar to those of 
server system 12. Workstation 18 is programmed with client system control software and 
GUI soflware. The GUI software generates and displays a Windows™-format GUI on 
monitor 20. Through the use of the GUI and user input 21, the pathologist or other 
clinician can select the specimen regions to be imaged by the microscope station 30 of a 
server system 12, and the desired magnification of the selected specimen region. Server 
system control commands, including the specimen region commands and magnification 
commands designating the image selected by the patholo^st, are generated by the client 
system control software and transmitted to the server system 12 over communication 
channel 16. The client system control software also processes the digital image data 
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received from the server system 12, and generates visual displays on monitor 20 of the 
associated specimen images represented by the data. 

Graphic User Interface And System Operation 
■ Figure 3 is an illustration of the GUI menu 50 generated by workstation 1 8 and 
displayed on monitor 20 at client system 14. The illustrated embodiment of menu 50 
includes a menu bar 52 and a toolbar 54. The pathologist operates system 10 by using 
keyboard 22 and/or mouse 24 in a conventional manner to select commands on menu bar 
52 and toolbar 54. As described below, other commands not graphicaUy displayed on the 
GUI can be selected by clicking the left and right buttons (not separately shov^) of the 
mouse 24. In addition to conventional Windows™ commands (e.g., Window and Help), 
menu bar 52 includes the system-specific commands System, Session and Microscope. 
Toolbar 54 includes a number of conventional Windows™ command buttons 55 - 58, 
microscope (e.g., stage drive) control buttons 59, and a new slide button 60. Buttons 55 
and 56 are format buttons for selecting tiled or cascaded specimen image displays, 
respectively. Arrange icon button 57 and information button 58 provide conventional 
Windows™ functions when selected. Get slide button 60 can be actuated by the 
pathologist to obtain a tiled slide overview image (described in greater detail below) when 
a new specimen slide has been positioned for imaging on the remote microscope station 
30. The tiled slide overview image can also be initiated by the technician at the server 
system 12. A listing of the commands and associated functions available through the 
selection of the System, Session, Microscope, Window and Help commands on menu bar 
52 follows. Many of the commands provide conventional Windows™-format functions, 
and are identified as "Standard" in the listing. 

Menu Bar Options 

System 


Preferaices 


Display "user preference" options to set parameters such as the way new 
images are displayed, the width of reticle borders, the way that gamma 
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correction is performed, and the server sites that are available 

Save 

Standard 

Save As 

Standard 

Print 

Standard 

Print Setup 

Standard 

Exit 

Standard 

Session 

Connect 

Establish diagnostic session with server system (remote microscope) site 

Gamma 
Correction 

Perform gamma correction on an existing image according to settings in 
"Preferences" menu 

Get Slide 

Retrieve overview of entire specimen 

Remote 
Database 

Establish session with existing remote database (as opposed to remote control 
of a microscope) 

Chat 

Establish TCP/IP "chat" session with remote node to interact with remote 
user via keyboard 

Submit 
Diagnosis 

Display form for pathologist to complete with info specific to this diagnostic 
session, including fields for diagnosis, confidence level, system performance, 
image quality, etc. then send completed form to microscope location to be 
included with 'trajectory" and patient history. This option finalizes the 
session 

Patient 
Information 

Display summary information retrieved from electronic patient record or 
entered by technician at microscope site 

Microscope 

Refocus 

Instruct remote microscope to perform autofocus routine (automatically 
retrieves another image) 

Intensity 

Instruct remote microscope to change illumination of specimen via neutral 
density filter and/or camera parameters 

Get Image 

Retrieve another copy of the current image fi*om r^ote microscope 

Image 

Perform interactive adjustment of image brightness to remotely calibrate the 
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Calibrate 

microscope's camera settings 

Window 


Tile 

Standard 


Cascade 

Standard 


Arrange All 

Standard 


About 

Standard 



Hdp 


Contents 

Standard 

Search for Help 

Standard 

How to Use Help 

Standard 


Figure 4 is an illustration of a GUI open session menu displayed on monitor 20 
and used by the pathologist to initiate a remote specimen observation session. The open 
session menu will be presented to the pathologist upon the selection of the Connect 
command available through menu bar 52. As shown, the open session menu includes a 
list of server systems 12 set up for operation v^th the client system 14, and displays the 
then-current specimen slide illumination (intensity) and image magnification (the 
established "user preference" values). Workstation 18 initiates and establishes an 
operational communication interface with the server system 12 selected by the pathologist 
through the open session menu. 

The pathologist will typically begin the diagnosis of a specimen by observing a 
slide overview image 70 such as that shown in Figure 5. The slide overview image 70 is a 
relatively low magnification image of a relatively large portion of the specimen on the 
slide, and in one embodiment is an image of the entire specimen on the slide. In the 
embodiment shown, the slide overview image 70 is an assembly of separate and discrete 
specimen region images 72 into a tiled overview image of the entire specimen on the slide. 
The slide overview image 70 shovm in Figure 5, for example, is formed from a 3 x 4 array 
of specimen re^on images 72. 
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Generation of the overview image 70 is controlled by the server system control 
software in workstation 26. In response to Get Slide commands (or Send Slide 
commands initiated by the clinician at server system 12), the workstation 26 actuates 
objective drive 38 to select a relatively low magnification (e.g., 2x) objective 36 and 
initiates an automatic focus routine. Stage drive 40 and camera 42 are then actuated to 
sequentially obtain the adjacent specimen region images 72. The image data for the 
specimen region images 72 is processed by the server system to reduce the resolution of 
the images, and transmitted to the client system 14 where it is assembled and displayed as 
the slide overview image 70. The image data for each of the specimen region images 72 
can be cached in memory of the workstation 18 so the image can be redisplayed as the 
topmost image on monitor 20, 

As noted above, the generation of a slide overview image 70 can be initiated at 
either the server system 12 or the client system 14, After positioning a new specimen 
slide on microscope station 30, the clinician can, for example, initiate the generation of a 
slide overview image 70 by selecting a Send Slide command through a graphic user 
interface and user input (not separately shown) at the server system 12. Image data 
representative of the slide overview image 70 will then be transmitted to the client system 
14 and displayed on monitor 20, thereby indicating to the pathologist the availability of 
the specimen slide for observation. If the image data has already been transmitted by the 
server system 12, the pathologist can also initiate the generation of a new slide ovenaew 
image 70 fi-om the client system through use of the Get Slide command button 60 on 
toolbar 54 or the Get Slide command available through menu bar 52. 

During observation of the slide overview image 70 the pathologist will typically 
identify specimen regions that he or she would like to observe in greater detail. Using the 
GUI and user input 21, the pathologist can select the specimen region of interest and the 
desired magnification at which the specimen region is to be observed. The appropriate 
server system control commands are then generated by the client system control software 
and transmitted to the server system 12 to initiate the retrieval and display of the specimen 
region of interest image (i.e., a detail image). 
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Specimen regions of interest can, for example, be selected by positioning the GUI 
cursor over the image and clicking the right button on mouse 24. As shown in Figure 6, 
this command causes a GUI microscope control menu 80 to be generated and displayed 
on monitor 20. Available commands in the microscope control menu 80 include Zoom 
and Zoom Factor. When the Zoom command is selected through user input 21 , the GUI 
software generates and displays on the then-displayed specimen region image an indicator 
such as reticle 82 shown in Figure 7. Using the user input 21 the pathologist can then 
reposition the reticle 82 over the specimen region of interest on the image, and thereby 
select the specimen region for further observation. Returning to the microscope control 
menu 80, the pathologist can then select the Zoom Factor command and cause the zoom 
factor menu 84 (Figure 6) to be generated and displayed by the GUI software. The zoom 
factor menu 84 includes a number of discrete magnification level commands 
corresponding to the available magnification levels (e.g., objectives 36) on microscope 
station 30, as well as In and Out step commands. The In command causes the 
magnification to increase one available power level fi-om the then-current magnification at 
the microscope station 30, while the Out command causes the magnification to decrease 
one available power level from the then-current magnification. By clicking the left button 
of mouse 24 after selecting the desired magnification and image region of interest in the 
manner described above, the pathologist will initiate the generation and transmission of 
the appropriate magnification and specimen region commands to retrieve the image. 

Using the client system 14 in the manner described above, the pathologist is able 
to relatively quickly observe images of many different specimen regions at different 
magnifications. As is evident from Figure 6, multiple images can be simultaneously 
displayed on monitor 20. The images can be displayed in a manner which enables the 
pathologist to easily identify the location of the specimen region image with respect to the 
image overview and with respect to all other images. The magnification of the image is 
also displayed. Workstation 26 automatically maintains a log or cache describing in 
chronological sequence the control commands transmitted to the server system 12 and the 
coordinates of the specimen region of the retrieved image. In addition to providing an 
audit trail, this information streamlines the image acquisition and transfer fiinctions and 
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provides the pathologist with immediate feedback regarding the regions of the specimen 
which have been imaged. Each of the retrieved specimen images can be stored by the 
workstation 26 for subsequent retrieval and observation. 

System 10 offers a number of important advantages, especially with respect to 
known "static" and "dynamic" architectures. High-resolution, diagnostic-quality, color 
images can be retrieved and displayed in real time. The system can be eflBciently operated 
through the user-friendly graphic user interface. Commercially available hardware can be 
used to implement the system. The network protocol is also robust, scalable (works on 
many different size channels), fast and relatively secure. The system allows pathologists 
to quickly provide high-quality diagnostic opinions to multiple remote locations, thereby 
making efficient use of their time. Accurate, precise, real-time control of the remote 
imaging system is achieved. The system can be integrated with electronic patient record 
systems to include patient information and diagnostic resuhs. The audit trails which can 
be provided are also advantageous. 

Virtual Focus Function 

Imaging system 10 can be configured to include a focal plane adjustment system 
which effectively provides a virtual focus capability to the pathologist at client system 14. 
The virtual focus function allows the pathologist to sequentially display on monitor 20 a 
series of different and discrete specimen re^on images, all of which are images of the 
same specimen region and at the same magnification, but which are taken at a range of 
focal planes on opposite sides of the "in-focus" focal plane to which the objective 36 is 
driven by the automatic focus system of the microscope 3 1 . The speed at which the 
different focal plane specimen region images are presented on monitor 20 can be 
controlled by the pathologist through the use of the GUI and/or user input 21 to simulate 
the observation of the specimen image under conventional manual focus control. 

The virtual focus function of system 10 can be selected by the pathologist through 
the use of the GUI and/or user interface 21 . One embodiment, for example, includes a 
Dynamic Focus command in the list of commands available under the Microscope 
command on menu bar 52. In response to pathologist selection of the Dynamic Focus 
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command, workstation 18 generates and transmits to server system 12 a focus adjust 
command. Generation of the diflferent focal plane specimen region images is controlled 
by the server system control software in workstation 26. In response to the receipt of the 
focus adjust command, workstation 26 actuates the automatic focus system of the 
microscope 3 1 and camera 42 to generate image data for several specimen region images 
taken at a range of diflferent focal planes on both sides of the in-focus focal plane 
determined by the automatic focus system of the microscope 3 1 . In one embodiment, for 
example, workstation 26 initiates the generation of three specimen region images at three 
diflferent focal planes below the in-focus focal plane, a specimen image at the in-focus 
focal plane, and three specimen region images at three diflferent focal planes above the in- 
focus focal plane. The selected "out-of-focus" focal planes are typically symmetrically 
located about the in-focus focal plane. The image data for the virtual focus specimen 
images is then transmitted to the client system 14. 

Upon receipt of the virtual focus specimen image data, the client system 14 will 
initially cause one of the images (typically the in-focus image) to be displayed on monitor 
20. The GUI presented with the virtual focus images will also include a focus control 
element which can be selected or actuated by the pathologist thr ough the use of user 
interface 21 . In one embodiment, for example, the GUI will include a conventional 
Window™-type slider bar on an edge of the display (not shown). Using the mouse 24 the 
pathologist can click on the arrows (also not shown) at the opposite sides of the slider 
bar, or click and drag on the slider button, to cause specimen region images at "adjacent" 
focal planes to be sequentially displayed on the monitor 20. 

During diagnostic sessions pathologists will routinely observe a given specimen 
region at a number of diflferent focal planes. The reason for these range of focus 
observations is due at least in part to the fact that the specimen being unaged has depth 
(i.e., is three-dimensional). Portions of the specimen at diflferent depths must therefore be 
imaged at diflferent focal planes. The virtual focus function of system 10 allows the system 
to closely emulate the manner by which a pathologist would analyze a specimen through 
the direct use of a conventional microscope. 
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Bandwidth Optimization Function 

System 10 also offers the pathologist the capability of selecting the fidelity of the 
specimen region images generated by server system 12 and transmitted to the client 
system 14 over communication channel 16. In one embodiment, for example, and as 
shovm in Figure 8, the GUI microscope control menu 80 generated and displayed on 
monitor 20 when the pathologist is selecting specimen regions of interest includes a 
Resolution command. When the Resolution command is selected through the use of the 
user input 21, a resolution menu 90 is generated and displayed on the monitor 20 by the 
GUI software. The resolution menu 90 includes a number of discrete, resolution level 
commands corresponding to the available resolution levels (e.g., bits per pixel for a full 
color image), as well as Better and Worse step commands. The Better command causes 
the resolution to increase one available level from the then-current resolution, while the 
Worse command causes the resolution to decrease one available level from the then- 
current resolution. By clicking the left button of mouse 24 after selecting the desired 
resolution, the pathologist will initiate the generation and transmission of the appropriate 
resolution commands to the server station 12. In response, the control software at the 
server station 12 controls the operation of workstation 26 in such a manner that the 
specimen image is incrementally transmitted until the selected resolution is attained. 

The capability of controlling the resolution of the specimen images generated and 
transmitted by the server system 12 provides important advantages. Images of specimen 
regions of particular interest to the pathologist can be generated and transmitted at 
relatively high fidelity (i.e., relatively low spatial and/or color space distortion). Lossless 
and high fidelity images of this type will enable the pathologist to provide accurate 
diagnostic opinions. Images of regions of lower priority, on the other hand, can be 
generated and transmitted at lower fidelity levels which require less bandwidth from the 
communication channel 16. Lower fidelity images can therefore be generated, transmitted 
and displayed with greater speed and efficiency, enhancing the overall efficiency and 
useftilness of the system 10. 

Additionally, if low-fidelity images have been transmitted and contain subregions 
which are of particular interest to the pathologist, the image information which is desired 


-16- 


wo 99/13360 


PCT/US98/18852 


for diagnostic confidence can be transmitted incrementally, for that subregion only. For 
example, the pathologist can be observing a full-screen, relatively low resolution image 
transmitted to client system 14 (e.g., through use of a Zoom command). If the image is 
largely uninteresting, but a relatively small subregion which could be important is 
identified, the pathologist can select the Resolution command in the manner described 
above. In response to the selection of the Resolution command, the GUI software 
generates and displays a resolution indicator such as a resolution reticle (not shown), 
which can be similar to the reticle 82 described above v^th respect to the Zoom 
command. The resolution reticle or other indicator can be positioned over the image 
subregion of interest using the GUI and user input 21 in a manner similar to that described 
above for manipulating Zoom command reticle 82. 

After the resolution reticle is positioned, the pathologist clicks the left button on 
mouse 24 to initiate the generation and transmission of a resolution command to the 
server system 12. The server system 12 has already generated the digital image data at 
full resolution (e.g., 24 bpp) for the entire image region being displayed at relatively low 
resolution on monitor 20, and "degraded" this image data to produce the previously 
transmitted relatively low resolution version. Server system 12 also includes a stored 
record identifying the image data previously transmitted for the relatively low resolution 
image. The server system 12 therefore transmits only the additional image data required 
to augment the already-transmitted image subregion with an incremental improvement to 
obtain the desired resolution. These Resolution command procedures can be repeated on 
the same (or overlapping) image subregions until the "incremental" image data 
transmissions have completed the total available resolution (e.g., 24 bpp) for the selected 
image subregion. When used in this manner, the Resolution command effectively 
provides a fianction similar to the Zoom command. 

Although the present invention has been described vAth reference to preferred 
embodiments, those skilled in the art will recognize that changes can be made in form and 
detail without departing fi-om the spirit and scope of the invention. In particular, although 
described as a telepathology system, the invention can be used in other remotely- 
controlled microscope applications. 
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WHAT IS CLAIMED IS: 

1 . A digital imaging system for enabling a person at a local site to observe 
and analyze images received in digital form from remote site, comprising: 
a server system at the remote site, including: 

image data means for providing digital image data representative of images; and 
a workstation interconnected to the image data means and including: 

a communication channel interface port adapted to be interfaced to a 

digital data communication channel over which the digital image 
data and server system control commands are transmitted 
between the local and remote sites; and 
image output means responsive to resolution commands for outputting 
the digital image data representative of images at selected 
resolutions; and 
a client system at the local site, including: 
a monitor; 

an operator-actuated user interface; and 

a workstation interconnected to the monitor and operator-actuated user interface 
and including: 

a communication channel interface port adapted to be interfaced to a 

digital data communication channel; 
image display means responsive to the digital image data received from 

the server system, for causing the monitor to visually display the 

images; and 

resolution command generation means for generating resolution 

commands, representative of the resolution of the digital image 
data to be transmitted to the client system by the server system, 
selected by the operator through use of the user interface. 
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2. The digital imaging system of claim 1 wherein the image data means of 
the server system workstation includes memory for storing digital image data 
representative of the images, 

3 . The digital imaging system of claim 2 wherein: 

the memory for storing digital image data representative of the images stores 

image data at a first resolution; and 
the resolution command generation means generates resolution commands, 

representative of the first resolution and at least a second resolution which 

is less than the first resolution, selected by the operator through use of the 

user interface. 

4. The digital imaging system of claim 1 wherein the workstation further 
includes graphic user interface display means for causing the monitor to display a graphic 
user interface enabling a person using the operator-actuated interface to control the server 
system , including selecting the resolution of the digital image data to be transmitted. 

5. The digital imaging system of claim 4 wherein: 

the graphic user interface display means of the workstation fiirther includes means 
for causing the monitor to display a graphic user interface enabling an 
operator using the operator-actuated user interface to select a specimen 
region of interest; and 

the resolution command generation means includes means for generating 

commands representative of the specimen region of interest and resolution 
of the digital image data selected by the operator through use of the 
graphic user interface display and the operator-actuated user interface. 

6. The digital imaging system of claim 4 wherein: 

the graphic user interface display means of the client system workstation fiirther 
includes means for causing the monitor to display a resolution menu in 
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response to clinician selection of a resolution command through use of the 
operator-actuated user interface; and 
the resolution command generation means of the client system workstation 
includes means for generating resolution commands in response to 
clinician selection of a desired resolution from the resolution menu 
through use of the operator-actuated user interface. 

7. The telemedicine imaging system of claim 6 wherein the resolution menu 
includes a list of resolution level commands representative of image data resolutions 
available from the server system. 

8. The telemedicine imaging system of claim 7 wherein the resolution menu 
includes step resolution level conunands representative of resolution increases and 
decreases to a next available resolution. 

9. The telemedicine imaging system of claim 4 wherein: 

the graphic user interface display means of the client system workstation includes 
means for causing the monitor to display a resolution region indicator on 
the specimen image in response to clinician selection of a resolution 
indicator command through use of the operator-actuated user interface, 
and wherein the indicator can be manipulated by the clinician through use 
of the operator-actuated user interface to designate a region of the image 
for a resolution change; 

the command generation means of the client system workstation includes means 
for generating resolution commands representative of the selected 
specimen image resolution for the designated region of the image; and 

the workstation of the server system includes resolution control means for causing 
the specimen image data outputted by the image output means to be 
representative of the selected specimen image resolution for the 
designated region of the image. 
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10. The digital imaging system of claim 1 wherein: 

the server system further includes an imaging instrument having a focus system, 
for imaging an object; 

the image data means of the server system includes a digital imaging system for 
providing distal image data representative of the images produced by the 
imaging instrument; 

the workstation of the server system is interconnected to the focus system and 
digital imaging system, and further includes focus system control means, 
responsive to dynamic focus commands, for controlling the focus system 
and causing the imaging instrument to image the object at a plurality of 
focal planes, wherein the image output means outputs digital unage data 
representative of the focal plane images produced by the instrument; 

the image display means of the client system workstation causes the monitor to 
visually display the plurality of focal plane images; and 

the client system workstation further includes dynamic focus command generation 
means for generating dynamic focus commands in response to actuation 
of the user interface. 

1 1 . The digital imaging system of claim 1 0 wherein the focus control means 
of the server system workstation causes the instrument focus system to image the object 
at an in-focus plane, at least one focal plane on a first side of the in-focus focal plane, and 
at least one focal plane on a second side of the in-focus focal plane, in response to the 
dynamic focus commands. 

12. The digital imaging system of claim 1 1 wherein the client system 
workstation further includes graphic user interface display means for causing the monitor 
to display a graphic user interface enabling a person using the operator-actuated user 
interface to select focal plane images for display. 
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13, A remotely-controlled digital imaging system for enabling an operator at a 
local site to observe images produced by an instrument at a remote site, including: 
a server system at a remote site, including: 

an imaging instrument having a focus system, for ima^g an object; 

a digital ima^g system for providing digital image data representative of the 
images produced by the imaging instrument; and 

a woiicstation interconnected to the focus system and digital imaging system, and 
having a communication channel interface port adapted to be interfaced to 
a digital data communication channel over which the digital image data 
and server system control commands, including dynamic focus 
commands, are transmitted between the local and remote sites, the 
workstation including: 

focus system control means, responsive to dynamic focus commands, for 
controlling the focus system and causing the imaging instrument 
to image the object at a plurality of focal planes; and 
image output means for outputting digital image data representative of the 
focal plane images produced by the instrument at the plurality of 
focal planes; and 
a client system at a local site, including: 
a monitor; 

an operator-actuated user interface; and 

a workstation interconnected to the monitor and operator-actuated user interface, 
and having a communication channel interface port adapted to be 
interfaced to the digital data communication channel, the workstation 
including: 

image display means responsive to the digital image data received from 
the server system, for causing the monitor to visually display the 
plurality of focal plane object images; and 

dynamic focus command generation means for generating dynamic focus 
commands in response to actuation of the user interface. 
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14. The digital imaging system of claim 13 wherein the focus system control 
means of the server system workstation causes the instrument focus system to image the 
object at an in-focus focal plane, at least one focal plane on a first side of the in-focus 
focal plane, and at least one focal plane on a second side of the in-focus focal plane, in 
response to the dynamic focus commands. 

15. The distal imaging system of claim 14 wherein the client system 
workstation further includes dynamic focus display control means responsive to the 
operator-actuated user interface, for causing the monitor to visually display the plurality 
of focal plane images in response to actuation of the user interface. 

16. The digital imaging system of claim 1 5 wherein the dynamic focus display 
control means includes graphic user interface display means for causing the monitor to 
display a graphic user interface enabling a person using the operator-actuated user 
interface to select focal plane images for display. 

17. The digital imaging system of claim 16 wherein the graphic user interface 
display means includes means for causing the monitor to display a graphic user interface 
enabling a person using the operator-actuated user interface to sequentially display the 
focal plane images. 

18. The digital imaging system of claim 1 7 wherein the graphic user interface 
display means includes means for causing the monitor to display a slider bar enabling a 
person using the operator-actuated user interface to sequentially display the focal plane 
images. 

19. The digital imaging system of claim 1 7 wherein the focus system of the 
imaging instrument includes an automatic focus control system. 
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20. The digital imaging system of claim 1 9 wherein the imaging instalment 
includes a microscope. 

21 . The digital imaging system of claim 1 3 wherein the client system 
workstation further includes graphic user interface display means for causing the monitor 
to display a graphic user interface enabling a person using the operator-actuated user 
interface to select focal plane images for display. 

22. The digital ima^g system of claim 21 wherein the graphic user interface 
display means includes means for causing the monitor to display a graphic user interface 
enabling a person using the operator-actuated user interface to sequentially display the 
focal plane images. 

23. The distal imaging system of claim 13 wherein: 

the client system workstation further includes graphic user interface display means 
for causing the monitor to display a menu having a dynamic focus 
command; and 

the dynamic focus command generation means includes means for generating 
dynamic focus commands in response to selection of the dynamic focus 
command from the menu through use of the user interface. 


-24- 


wo 99/13360 


PCT/US98/18852 




2/7 


wo 99/13360 PCT/US98/18852 


^TelePath Client 


System Se^iion Ulacscope W?nd6w Help 


:0: 






A 

V 


Open Session Dialog x 

Select Microscope Location: 

Startup 

Intensity; 4§ 
Objective: 1 § 

Center for Teleconnunlcations 

Springbrook Hospital 
Solo Memorial ot Skycity 
Cliffline Healtti Center 


✓ OK 1 

X concel ? Help j 



3/7 


wo 99/13360 


PCT/US98/18852 


^TeiePoth Client 

itapi rn i<{|>iAiv 


Slide Overview 


Em 


•..V 


, * t 



r 


z 


7^7 


7Z 


4/7 


wo 99/13360 PCT/US98/1 8852 


i^TelePom Client 
Syttem Sewion td'cro-'cope Window Help 


:B; 


/ (24 zg 16293) ObtecUve-. Z* f iKgrHtS 




(207e. 17 ]B4} OPj^}^: tOx Rterj(f.'5 

# ' r • 










Zoom 1 y 

' * • •* * 


Zoom Foctor 

In 

' * . ' * * 
i. * : ' ^, 

s • *. • 
. i • 


Refocus 
Copy 

Save imoqe 

Out 
2x 
4x 
lOx 



• • • • • 

** * *. 

20 X 
40x 


,»,*•• • * 


lOOx 

J' 



* ** .* 

* ' . • * » » 

.»■ • ." " ..' ' ' 


r * 



V • 



■ - • .r.A • • • • 






5/7 


wo 99/13360 


PCT/US98/18852 


i^TelePoih Client 


Sytlem Sejjion (^Ktosczz^ Window Help 


:0; 



5 


< 

> 

A 

V 


4T-T 


513 


4-- ' ' 


. - v. . . ..• 



A 1 


c 


6/7 


PCT/US98/18852 


80' 


Zoom 

Zoom Eoctor 


. •Resblutidh; -v^ 

Better 
Worse 
2 bpp 
4 bpp 
8 bpp 
24 bpp 

Refocus 
Copy 

S_ave Image 



111 


4 a '■^ 
« ,. . . 


anhang zum europalschen recherchenbericht 
Ober die europAische patentanmeldung nr. 


EP 01 11 5337 


In diesem Anhang sind die Mrtglieder der Patentfamilien der im obengenannten europaisohen Recherchenbericht angefOhrten 
Patentdokumente angegeben. 

Die Angaben Qber die Familienmitglieder entsprechen dem Stand der Datei des Europdtschen Patentamts am 
Diese Angaben dienen nur zur Unterriohtung und erfolgen ohne Gewahr. 

22-06-2004 


Im Recherchenbericht 
angefOhrtes Patentdokument 

Datum der 
Vereffentlichung 

Mitglied(er)der 
Patentfamilie 

Datum der 
VerOffentlichung 

DE 19633997 

C 

26-03-1998 

DE 

19633997 CI 

9^.^7.1 QQA 




AU 

718452 B2 





AU 

4118897 A 





CA 






WO 

9808342 Al 

oc n9 1QQA 




EP 


09-06-1999 




JP 

2000516784 T 

12-12-2000 




US 

6650703 Bl 

18-11-2003 

WO 9913360 

A 

18-03-1999 

AU 

9476798 A 

9Q.n'^.1 QQQ 




WO 

9913360 A2 

IP no 1QQQ 

WO 9801999 

A 

15-01-1998 

AT 

210357 T 

10- ic-tUUi 




AU 

3559297 A 

02-02-1998 




DE 

69708871 Dl 

17-01-2002 




DE 

69708871 T2 

01-08-2002 




DK 

910921 T3 

02-04-2002 




EP 

0910921 Al 

28-04-1999 




ES 

2171266 T3 

01-09-2002 




H 

0 

986121 A 

10-03-1999 




WO 

9801999 Al , 

15-01-1998 




PT 

910921 T 

31-05-2002 




US 

6208374 Bl 

27-03-2001 

EP 0994433 

A 

19-04-2000 

EP 

0994433 Al 

19-04-2000 

US 5216596 

A 

01-06-1993 

AU 

5435790 A 

16-11-1990 




WO 

9012627 Al 

01-11-1990 




US 

5297034 A 

22-03-1994 




EP 

0293083 A2 

30-11-1988 




JP 

1020449 A 

24-01-1989 



FQr nahere Einzelheiten zu diesem Anhang : siehe Amtsbtatt des Europaisohen Patentamts, Nr.12/82 


Nummer der Anmeldung 

EP Gl 11 5337 


einschlAgige dokumente 


Kategorie 


Kennzeichnung des Dokuments mit Angabe, soweit eriorderlich, 
der maBgeblichen Telle 


Betrifft 
Anspruch 


KLASSIRKATION DER 
ANMELDUNG (lnt.CI.7) 


A.D 


DE 196 33 997 C (UNIV STUTTGART) 
26. Marz 1998 (1998-03-26) 

* Spalte 1, Zeile 3 - Spalte 3, Zeile 47 * 

* Spalte 8, Zeile 21 - Zeile 56 * 

WO 99/13360 A (BELLSOUTH INTELLECT PTY 
CORP) 18. Marz 1999 (1999-03-18) 

* Zusamnenfassung * 

* Seite 16, Zeile 1 - Seite 17, Zeile 11 * 

WO 98/01999 A (CLINCH NOAH ;SECOND OPINION 

SOLUTIONS AS (NO)) 

15. Januar 1998 (1998-01-15) 

* Seite 3, Zeile 21 - Seite 4, Zeile 1 * 

EP 0 994 433 A (FAIRFIELD IMAGING LTD) 
19. April 2000 (2000-04-19) 

* Absatz [0028] * 

US 5 216 596 A (WEINSTEIN RONALD S) 
1. Juni 1993 (1993-06-01) 

* Spalte 2, Zeile 16-53 * 


1-10 


1.6 


1.6 


1.6 


1.6 


GO2B21/00 


RECHERCHIERTE 
SACHGEBIETE (lnl.CI.7) 


G02B 

H04N 
G06F 
G06T 


Der vorliegende Recherche nbericht wurde fOr alle PatentansprQche erstellt 


Recherchenort 

BERLIN 


AbschtuBdatum der Recherche 

22. Juni 2004 


PrOfer 

Verdrager, V 


KATEGORIE DERGENANNTEN DOKUMENTE 

X : von besonderer Bedeutung allein betrachtet 

Y : von besonderer Bedeutung in Verbindung mit einer 

onderen Ver6ffentiichung derselben Kategorie 
A : techno log ischer Hintergmnd 
O : njchtftchrtftliche Offenbarung 
P ; Zwtschenliteratur 


T : der Erfindung zugru nde liege nde Theorien oder Grundsfltze 
E : altered Patentdokument, das jedoch erst am oder 
nach dem Anmeldedatum verOffentlicht worden iat 
D : in der Anmeldung angefOhrtes Dokument 
L : au8 anderen GrQnden angefQhrtea Dokument 

& : Mitgtied dergleichen Patentfamllie.Qbereinstimmendea 
Dokument 


